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(54) TiUe: A SPEECH INTERFACE FOR AN AUTOMATED ENDOSCOPIC SYSTEM 
(57) Abstract 

A robotic system (10) which controls the movement of a surgical 
instrument (16) in response to voice conunands from die user. The 
robotic system (10) has a computer controlled aim (14) that holds the 
surgical instrument (16). The user provides voice commands to the 
conqmter (20) duough a microphone (40). The computer (20) contahis 
a phrase recognizer diat matches the user's speech widi words stored 
in die computer (20). Matched words are dien processed to detennine 
whedier die user has spoken a robot command. If die user has spoken a 
recognized robot command die computer (20) will move die robotic arm 
(14) in accordance with the command. 
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A SPEECH INTERFACE FOR AN AUTOMATED 
ENDOSCOPIC SYSTEM 

BACKGROUND OF THE INVENTION 

1. FIELD OF THE INVENTION 

The present invention relates to a robotic system that moves a 
surgical instrument in response to voice commands from the user. 

2. DESCRIPTION OF RELATED ART 

To reduce the invasiveness of surgery, endoscopes are commonly 
utilized to view the internal organs of a patient One end of the 
endoscope contains a lens which is inserted into the patient through a 
small incision in the skin. The lens focuses an image that is transmitted 
by fiber optic cable to a camera located at the opposite end of the 
endoscope. The camera is coupled to a monitor that displays the image 
of the patient. 

The endoscope can be used in cox\junction with another surgical 
instrument that is inserted into the patient. An assistant typically hold 
the endoscope while the surgeon manipulates the surgical instrument. 
The assistant moves the endoscope in response to instructions from the 
sxu*geon. Any miscommunication between the surgeon and the 
assistant may result in an error in the movement of the endoscope, 
thereby requiring the surgeon to repeat the mstruction. Additionally, 
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holding the endoscope for a significant amount of time may cause the 
assistant to become fatigued. 

U.S. Application Serial No. 07/927,801 discloses a robotic arm 
that holds and moves an endoscope. The surgeon can move the robotic 
arm by depressing a foot pedal. The foot pedal is connected to a 
computer which moves the arm and the scope. Although the '801 
system effectively moves the endoscope, the surgeon must continually 
manipulate the foot pedal, a process which may detract from the 
surgical procedure. It would be desirable to provide a robotic endoscopic 
system that can be controlled by voice commands from the user. 
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RTTMMARY OF THE INVENTION 

The present invention is a robotic system which controls the 
movement of a surgical instrument in response to voice commands firom 
the user. The robotic system has a computer controlled arm that holds 
the surgical instrument. The user provides voice commands to the 
computer through a microphone. The computer contains a phrase 
recognizer that matches the user's speech with words stored in the 
computer. Matched words are then processed to determine whether the 
user has spoken a robot command. If the user has spoken a recognized 
robot command the computer will move the robotic arm in accordance 
with the command. 
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BRTEF DESCRIPTION OF THE DRAWINGS 

The objects and advantages of the present invention will become 
more readily apparent to those ordinarily skilled in the art after 
reviewing the following detailed description and accompanying drawings, 
wherein: 

Figure 1 is a perspective view of a robotic endoscope system of 
the present invention; 

Figure 2 is a schematic of an endoscope within two separate 
coordinate systems; 

Figure 3 is a top view of a foot pedal; 

Figure 4 is a schematic of a computer system; 

Figure 5 is a schematic of a grammar process; 

Figure 6 is a schematic of a robotic arm. 
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nigTAn.ED DKSCRTPTIQN OF THE INVENTION 

Rrferring to the drawings more particularly by reference 
numbers^ Figure 1 shows a robotic system 10 of the present invention. 
The system 10 is typically used in a sterile operating room where a 
surgeon performs a surgical procedure on a patient. The patient is 
placed on a operating table 12. Attached to the table 12 is a robotic arm 
assembly 14 which can move a surgical instrument 16 relative to the 
table 12 and the patient. The surgical instrument 16 is typically an 
endoscope which is inserted into the abdomen of the patient 12. The 
endoscope 16 enters the patient through a canjiula, wherein the scope 
16 rotate about a cannula pivot point. The endoscope is typically 
connected to a monitor 18 which allows the surgeon to view the organs, 
etc. of the patient. Although an endoscope is described and shown, it is 
to be imderstood that the present invention can be used with other 
surgical instruments. 

The robotic arm assembly 14 controlled by a computer 20. In the 
preferred embodiment, the robotic arm assembly 16 includes a linear 
actuator 24 fixed to the table 14. The linear actuator 24 is connected to 
a linkage arm assembly 26 and adapted to move the linkage assembly 
26 along the z axis of a first coordinate system. The first coordinate 
system also has an x axis and a y axis. 

The linkage arm assembly 26 includes a first linkage arm 28 
attached to a first rotaiy actuator 30 and an end effector 32. The first 
rotaiy actuator 30 is adapted to rotate the first Unkage arm 28 and end 
effector 32 in a plane perpendicular to the z axis (x-y plane). The first 
rotary actuator 30 is connected to a second rotaiy actuator 34 by a 
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second linkage arm 36. The second actuator 34 is adapted to rotate the 
first actuator 30 in the x-y plane. The second rotary actuator 34 is 
connected to the output shaft of the linear actuator 24. The actuators 
24, 30 and 34 rotate in response to output signals provided by the 
computer 20. As shown in Figure 2, the junction of the endoscope 16 
and the end efifector 32 define a second coordinate system which has an 
x' axiSt a y' axis and a z' axis. The jtmction of the end efifector 32 and 
endoscope 18 also define the origin of a third coordinate system which 
has a x" axis, a y" axis and a z" axis. The z*' axis parallel with the 
longitudinal axis of the endoscope 16. 

The arm assembly may have a pair of passive joints that allow 
the end effector to be rotated in the direction indicated by the arrows. 
The actuators 24, 30 and 34, and joints of the arm may each have 
position sensors (not shown) that are connected to the computer 20. 
The sensors provide positional feedback signals of each corresponding 
arm component. 

The system has a microphone 40 that is connected to the 
computer 20. The system may also have a speaker 42 that is 
connected to the computer 20. The microphone 40 and speaker 42 may 
be mounted to a headset 44 that is worn by the user. Placing the 
microphone 40 in close proximity to the user reduces the amoimt of 
backgrotmd noise provided to the computer and decreases the 
probability of an inadvertent input command. 

As shown in Figure 3, the system may also have a foot pedal 50. 
The foot pedal 22 has a housing 56 that supports a pair of outer first 
foot switches 58 and a second foot switch 60. One outer foot switch 58 
has a first pressure transducer 62 and the other switch has a second 
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pressvire transducer 64. The second foot switch 60 has third 66, fourth 
68» fifth 70 and sixth 72 pressure transducers. The transducers are 
each connected to a corresponding operational amplifier that provides a 
voltage input to the computer 20. The pressure transducers 62-72 are 
preferably constructed so that the resistance of each transducer 
decreases as the surgeon increases the pressure on the foot switches. 
Such a transducer is sold by Interlink Electronics. The decreasing 
transducer resistance increases the input voltage provided to the 
computer 20 fi:xim the operational amplifier. Each transducer 
corresponds to a predetermined direction within the image displayed by 
the monitor. In the preferred embodiment, the first pressure transducer 
62 corresponds to moving the endoscope toward the image viewed by the 
surgeon. The second transducer 64 moves the scope away fi'om the 
image. The third 66 and fourth 68 transducers move the image **up*' and 
"down", respectively, and the fifth 70 and sixth 72 transducers move the 
image "left" and "right", respectively. The pedal may have a button 73 
that enables the foot pedal 50 and disable the voice command feature, or 
vice versa. 

Figure 4 shows a schematic of the computer 20. The computer 
20 has a multiplexer 74 which is connected to the pressure transducers 
of the foot pedal 50 and the position sensors of the arm. The multiplexer 
74 is connected to a single analog to digital (A/D) converter 76. The 
computer 20 also has a processor 78 and memory 80. 

The processor 78 is connected to an address decoder 82 and 
separate digital to analog (D/A) converters 84. Each D/A converter is 
connected to an actuator 24, 30 and 34. The D/A converters 84 provide 
analog output signals to the actuators in response to output signals 
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Teceived from the processor 78. The analog output signals have a 
sufficient voltage level to energize the electric motors and move the 
robotic arm assembly. The decoder 82 correlates the addresses provided 
by the processor with a corresponding D/A converter, so that the correct 
motor(s) is driven. The address decoder 82 also provides an address for 
the input data from the A/D converter 76 so that the data is associated 
with the correct input channel. 

The computer 20 has a phrase recognizer 86 connected to the 
microphone 40 and the processor 78. The phrase recognizer 86 digitizes 
voice commands provided by the user through the microphone 40. The 
voice commands are then processed to convert the spoken words into 
electronic form. The electronic words are typically generated by 
matching the user's speech with words stored within the computer 20. 
In the preferred embodiment, the recognizer 86 is an electronic board 
with accompanying software that is marketed by Scott Instruments of 
Denton, Texas under the trademark "Coretechs Technology". 

The electronic words are provided to the processor 78. The 
processor 78 compares a word, or a combination of words to predefined 
robot commands that are stored within a libraiy in the memoiy 80 of 
the computer 20. If a word, or combination of words match a word or 
combination of words in the library, the processor 78 provides output 
commands to the D/A converter 84 to move the robotic arm in 
accordance with the command. 

Rgure 5 shows ex^plary words and combinations of words that 
provide robot commands. A grammar process is performed to determine 
whether the voice commands satisfy certain conditions. The process 
contains a nimiber of states advanced by the satisfaction of a condition. 



wo 96/09587 



PCT/US95/12064 



-9- 

If the voice command provided by the user satisfies a first condition, 
then the process proceeds to the first state. If a condition of a next 
state is satisfied then the process proceeds to the next corresponding 
state, and so forth and so on. For example, to prevent a robot command 
from being inadvertently spoken, it is desirable to predicate all voice 
commands with a qualifier. For example, the qualifier may be a name 
given to the robot such as "AESOP". Therefore when the user provides 
a voice command, the process initially determines whether the spoken 
word is AESOR If the spoken word is not AESOP then the process 
ends. The term "stop" may be an exception to this rule, wherein the 
computer will stop arm movement when the user provides a simple 
"stop" voice command. 

If the spoken word is AESOP the process continues to state 1. 
The process next determines whether the user has spoken a word that 
satisfies a condition to advance to states 2-6. These words include 
"move", "step", "save", "return", "speed", "track instrument" and "track 
head". The track instrument command is for a system which has the 
ability to move an endoscope to automatically track the movement of a 
second instrument that is inserted into the patient. The track head 
command may enable the system so that the endoscope movement 
tracks the user's eyes. For example, if the user looks to the right of the 
image displayed by the monitor, the robot will move the endoscope to 
move the image in a rightward direction. The move and step conmiands 
induce movement of the scope in a desired direction. The save command 
saves the position of the endoscope within the memory of the computer. 
The return command will return the scope to a saved position. 



wo 96/09587 



PCT/US95/12064 



-10- 

From states 2-6 the process will determine whether the user has 
spoken words that meet the next condition and so forth and so on. When 
a certain numher of conditions have been met, the procMSor 78 wiU 
provide an output command to the D/A converter 84 in accordance with 
the voice commands. For example, if the user says "AESOP move left", 
the processor 78 will provide output commands to move the endoscope 
12, so that the image displayed by the monitor moves in a leftward 
direction. The microphone 40 phrase recognizer 86 and grammar 
process essentially provide the same input ftmction as the foot pedal 50, 
mviltiplexer 74 and A/D converter 76. 

The processor 78 can also provide the user with feedback 
regarding the recognized command through the speaker 42 or the 
monitor 18. For example, when the user states "AESOP move right", 
afl«r processing the speech, the processor 78 can provide an audio 
message through the speaker 42, or a visual message on the monitor 18, 
"AESOP move right". Additionally, the processor 78 can provide 
messages regarding system errors, or the present state of the system 
such as "speed is set for slow". 

Referring to Figure 6, the processor 78 typically computes the 
movement of the robotic arm assembly 16 in accordance with the 
following eqtiations. 

<1> ^3'^-^°^ t~^2Lil2 J 



wo 96/09587 



PCTAJS95/12064 



-11- 

a2 = flO+/-A 

angle between the second linkage arm 36 and the x axis, 
angle between the first linkage arm 28 and the 
longitudinal axis of the second linkage arm 36. 
length of the second linkage arm. 
length of the first linkage arm. 
x coordinate of the end effector in the first coordinate 
system. 

y coordinate of the end effector in the first coordinate 
system. 

To move the end effector to a new location of the x-y plane the processor 
78 computes the change in angles a2 and a3 and then provides output 
signals to move the actuators accordingly. The original angular position 
of the end effector is provided to the processor 78 by the position 
sensors. The processor moves the linkage arms an angle that . 
corresponds to the difference between the new location and the original 
location of the end effector. A differential angle ^^2 corresponds to the 
amount of angular displacement provided by the second actuator 34» a 
differential angle ^a3 corresponds to the amount of angular 
displacement provided by the first actuator 30. 

To improve the effectiveness of the system 10» the system is 
constructed so that the desired movement of the surgical instrument 
correlates to a direction relative to the image displayed by the monitor. 



where; 

a3 = 

Ll = 
L2 = 
x = 

y = 
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Thus when the surgeon commands the scope to move up, the scope 
alwajrs appears to move in the up direction. To accomplish this result, 
the processor 78 converts the desired movemrat of the end of the 
endoscope in the third coordinate system to coordinates in the second 
coordinate system, and then converts the coordinates of the second 
coordinate system into the coordinates of the first coordinate system. 

Referring to Fig. 2, the desired movement of the endoscope is 
converted from the third coordinate system to the second coordinate 
system by using the following transformation matrix; 



(2) 





f 


Ay 









cos(a6) 0 -sin(fl6) 

-sin(aS)sin(a6) cos(a5) -sin(a5)cos(fl6) 
(aS)sin(a6) sui(aS) cos(aS)cos(<i6) ; 



Ay 



where; 



Ax" = the desired incremental movement of the scope along 

the axis of the third coordinate system. 
Ay " = the desired incremental movement of the scope along 

the y" axis of the third coordinate system. 
Ae * = the desired incremental movement of the scope along 

the z" axis of the third coordinate system. 
a5 = the angle between the z* axis and the scope in the y -z 

plane. 

a6 = the angle between the z* axis and the scope in the x -z* 
plane. 

Ax' =: the computed incremental movement of the scope along 
the x' axis of the second coordinate system. 
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Ay' the computed incremental movement of the scope along 
the y' axis of the second coordinate system. 

Az' = the computed incremental movement of the scope along 
the z* axis of the second coordinate system. 



The angles a5 and a6 are provided by position sensors located on the end 
effector 32. The angles a5 and a6 are shown in Fig. 2. 

The desired movement of the endoscope is converted from the 
second coordinate system to the first coordinate system by using the 
following transformation matrix; 









''cos(;r) 


-sin(;r) 


0> 






(3) 


Ay 




sin(;r) 


cos(a) 


0 




Ay 








. 0 


0 


h 







where; 

Ax' = the computed incremental movement of the scope along 

the x' axis of the second coordinate system. 
Ay' = the computed incremental movement of the scope along 

the y' axis of the second coordinate system. 
Az* = the computed incremental movement of the scope along 

the z* axis of the second coordinate system. 
'n= is the angle between the first linkage arm and the x axis 

of the first coordinate system. 
Ax = the computed incremental movement of the scope along 

the x axis of the first coordinate system. 
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Ay s the computed incremental movement of the scope along 
the y axis of the first coordinate system. 

Az s the computed incremental movement of the scope along 
the z axis of the first coordinate system. 

The incremental movements Ax and Ay are inserted into the algorithms 
described above for computing the angular movements ( A a2 and Aa3) 
of the robotic arm assembly to determine the amount of rotation that is 
to be provided by each electric motor. The value Az is used to determine 
the amount of linecir movement provided by the linear actuator 24. 

The surgical instrument is t3Tpically coupled to a camera and a 
viewing screen so that any spinning of the instrument about its own 
longitudinal axis will result in a corresponding rotation of the image on 
the viewing screen. Rotation of the instrument and viewing image may 
disorient the viewer. It is therefore desirable to maintain the orientation 
of the viewing image. In the preferred embodiment, the end eflfector has 
a worm gear (not shown) which rotates the surgical instrument about 
the longitudinal axis of the instrument. To insure proper orientation of 
the endoscope 16, the worm gear rotates the instrument 16 about its 
longitudinal axis an amount Dq6 to insure that the y" axis is oriented in 
the most vertical direction within the fixed coordinate system. Dq6 is 
computed firom the following cross-products. 



Dq6 = zi"x(yo"xyi") 
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where; 

Dq6 = the angle that the instrument is to be rotated about the z" 

axis. 

yo'* = is the vector orientation of the y'* axis when the instrument 
is in the first position. 

yi" = is the vector orientation of the y" axis when the instrument 
is in the second position. 

zi" = is the vector orientation of the z** axis when the instrument 
is in the second position. 

The vectors of the yi" and zi" axis are computed with the following 
algorithms. 





C0S£}6 


0 


-sina6 




"0" 




-sina5sina6 


cosa5 


-sinfl5cosa6 




0 




_ cosa5sina6 


sina5 


cos^Scosa6 ^ 




1^ 



yi"=zrxxr 

where; 

aS = is the angle between the instrument and the z axis in the y-z 

plane. 

a6 = is the angle between the instrument and the z axis in the x-z 

plane. 

z = is the unit vector of the z axis in the first coordinate system. 
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The angles aS and a6 are provided position sensors. The 
vector yo" is computed ttsing the angles a5 and a6 of the instrument in 
the original or first position. For the computation of yi" the angles a5 
and a6 of the second position are used in the transformation matrix. 
After each arm movement yo" is set to yi" and a new yi" vector and 
corresponding Dq6 angle are computed and used to re-orient the 
endoscope. Using the above described algorithms, the worm gear 
continuously rotates the instrument about its longitudinal axis to insure 
that the pivotal movement of the endoscope does not cause a 
corresponding rotation of the viewing image. 

The system may have a memory feature to store desired 
instrument positions within the patient. The memory feature may be 
enabled either by voice commands or through a button on an input 
device such as the foot pedal. When a save command is spoken, the 
coordinates of the end effector in the first coordinate system are saved 
in a dedicated address(es) of the computer memory. When a return 
command is spoken, the processor retrieves the data stored in memory 
and moves the end effector to the coordinates of the effector when the 
save commemd was enabled. 

The memory feature allows the operator to store the coordinates 
of the end effector in a first position, move the end effector to a second 
position and then return to the first position with a simple command. 
By way of example, the surgeon may take a wide eye view of the patient 
firom a predetermined location and store the coordinates of that location 
in memory. Subsequently, the surgeon may manipulate the endoscope 
to enter cavities, etc. which provide a more narrow view. The surgeon 
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can rapidly move back to the wide eye view by merely stating "AESOP 
return to one". 

In operation, the user provides spoken words to the microphone. 
The phrase recognizer 86 matches the user's speech with stored words 
and provides matched electronic words to the processor 78. The 
processor performs a grammar process to determine whether the 
spoken words are robot commands. If the words are commands, the 
computer energizes the actuators and moves the endoscope, 
accordingly. The system also allows the user to control the movement 
of the endoscope with a foot pedal if voice commands are not desired. 

While certain exemplary embodiments have been described and 
shown in the accompanying drawings, it is to be understood that such 
embodiments are merely illustrative of and not restrictive on the broad 
invention, and that this invention not be limited to the specific 
constructions and arrangements shown and described, since various 
other modifications may occur to those ordinarily skilled in the art. 
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What is claimed is: 

1. A robotic system that controls a sxirgical instrument, 
comprising: 

a mechanism that can move the surgical instrument; 

an audio input device that receives a voice command from a user 
to move the surgical instrument, and provides an output signal that 
correspond to the voice command; and, 

a controller that receives said output signal from said audio input 
device and provides a movement output signal to said mechanism to 
move said mechanism and the surgical instrument in accordance with 
the voice command. 

2. The system as redted in daim 1, wherein said controller 
provides said movement output signal only when a plurality of voice 
conunands are provided in a predetermined sequence. 

3. The system as recited in daim 1, wherein said mechanism is a 
robotic arm assembly. 

4. The system as recited in daim 1, further comprising a foot 
pedal that is coupled to said controller. 

5. The ^stem as redted in daim 1, further comprising a speaker 
coupled to said controller to generate an audio message. 



wo 96/09587 



PCTAJS9Sa2064 



•19- 

6. The sysbem as recited in claim 1, fixrther compriBing a monitor 
coupled to said controller to generate a visual message. 

7. A robotic aystem that controls a surgical instrument, 
comprising: 

a mechanism that can move the surgical instrument; 

audio input means for receiving a voice command from a user to 
move the surgical instrument and providing an output signal that 
correspond to the voice command; axid» 

controller means for receiving said output signal from said audio 
input means and providing a movement output signal to said 
mechanism to move said mechanism and the surgical instrument in 
accordance with the voice command. 

8. The system as recited in claim 7, wherein said controller 
means provides said movement output signal only when a plurality of 
voice commands are provided in a predetermined sequence. 

9. The system as recited in claim 7, wherein said mechanism is a 
robotic arm assembly. 

10. The system as recited in claim 7, fiirtiier comprising a foot 
pedal this is coupled to said controller means. 

11. The sjrstem as redted in daim 7, further comprising speaker 
means for generating an audio message. 
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12« The system as recited in claim 7, further comprising monitor 
means for generating a visual message. 

13. A method for moving a surgical instrument, comprising the 
steps of: 

a) generating a voice command to move a mechanism that 
holds the surgical instrument; 

b) comparing the voice command with a robot command; and, 

c ) moving the surgical instrument with said mechanism if the 
voice command is a robot command. 

14, The method as recited in daim 13, wherein the surgical 
instrument is moved only when a pltirality of voice commands are 
provided in a predetermined sequence. 
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